Ultrasound-guided medical tool insertion simulators

ABSTRACT

A medical tool insertion simulation apparatus. The medical tool insertion simulation apparatus may comprise a syringe having an injection end, and an elongated member protruding from the injection end of the syringe. The elongated member may be configured to decrease a length of protrusion of the elongated member from the injection end of the syringe. The elongated member may be configured to receive at least one first sensor configured to indicate position information relating to the elongated member. At least one computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method for simulating medical tool insertion. A medical tool insertion simulation system.

RELATED APPLICATIONS

This application is a continuation-in-part of International ApplicationNo. PCT/US2016/051904, filed on Sep. 15, 2016, which claims the benefitunder 35 U.S.C. § 119(e) of U.S. Provisional Patent Application Ser. No.62/219,050, filed on Sep. 15, 2015, under Attorney Docket No.U1196.70004US00 and entitled “ULTRASOUND-GUIDED MEDICAL TOOL INSERTIONSIMULATORS,” and this application is a U.S. National Stage applicationunder 35 U.S.C. § 371 based on International Application No.PCT/US2016/051899, filed on Sep. 15, 2016, which claims the benefitunder 35 U.S.C. § 119(e) of U.S. Provisional Patent Application Ser. No.62/219,045, filed on Sep. 15, 2015, under Attorney Docket No.U1196.70003US00 and entitled “ULTRASOUND-GUIDED MEDICAL TOOL INSERTIONSIMULATORS,” each of which is hereby incorporated herein by reference inits entirety.

BACKGROUND

A simulation provides representations of certain key characteristics orbehaviors of a selected physical or abstract system. Simulations can beused to show the effects of particular courses of action. A physicalsimulation is a simulation in which physical objects are substituted fora real thing or entity. Physical simulations are often used ininteractive simulations involving a human operator for educationaland/or training purposes. For example, mannequin patient simulators areused in the healthcare field, flight simulators and driving simulatorsare used in various industries, and tank simulators may be used inmilitary training.

Physical simulations or objects provide a real tactile and hapticfeedback for a human operator and a 3-dimensional (3D) interactionperspective suited for learning psycho-motor and spatial skills. In thehealth care industry, as an example, medical simulators are beingdeveloped to teach therapeutic and diagnostic procedures, medicalconcepts, and decision making skills. Many medical simulators involve acomputer or processor connected to a physical representation of apatient, such as a mannequin patient simulator.

Virtual simulations have also been used for education and training.Typically, the simulation model is instantiated via a display such as acomputer, PDA, or cell phone screen; or a stereoscopic, 3D, holographic,or panoramic display. An intermediary device, often a mouse or joystick,may be needed to interact with the simulation.

The use of ultrasound in medicine is becoming the standard of care formany procedures. Procedures that utilize ultrasound range from fetalimaging to the safe insertion and guidance of a needle inside a patient.Ultrasound uses sound waves to generate a 2-dimensional (2D) (or in somemachines, a 3D) image of underlying muscle, tissue, bone, and otherstructures.

SUMMARY

Some aspects include a medical tool insertion simulation apparatus. Themedical tool insertion simulation apparatus may comprise a syringehaving an injection end, and an elongated member protruding from theinjection end of the syringe. The elongated member may be configured todecrease a length of protrusion of the elongated member from theinjection end of the syringe. The elongated member may be configured toreceive at least one first sensor configured to indicate positioninformation relating to the elongated member.

Further aspects include at least one computer-readable storage mediumencoded with executable instructions that, when executed by at least oneprocessor, cause the at least one processor to perform a method forsimulating medical tool insertion. The method may comprise: receivingultrasound data representing an object being imaged; receiving positiondata from a hand-held device; computing a position of a medical toolsimulator attached to the hand-held device; and generating an image ofthe object with an image of a simulated medical tool positioned relativeto the object based on the computed position of the medical toolsimulator.

Additional aspects include a medical tool insertion simulation system.The medical tool insertion simulation system may comprise a medical toolsimulator configured to decrease a length of protrusion of a protrudingmember of the medical tool simulator; and an ultrasound probe. Themedical tool simulator may be configured to receive at least one firstsensor. The ultrasound probe may be configured to receive at least onesecond sensor. The at least one first sensor and/or the at least onesecond sensor may be configured to indicate a relative position betweenthe medical tool simulator and the ultrasound probe.

Further aspects include at least one computer-readable storage mediumencoded with executable instructions that, when executed by at least oneprocessor, cause the at least one processor to perform a method forsimulating medical tool insertion. The method may comprise: presenting auser interface representing a simulated medical tool on a display of thefirst device; receiving, from a first sensor of the first device, anorientation of the first device; and transmitting, to a second device,the orientation of the first device.

In some embodiments, the method may further comprise: receiving, via theuser interface, a sliding user input representing a depth of insertionof the simulated medical tool; and transmitting, to the second device,the sliding user input. Alternatively or additionally, the method mayfurther comprise: receiving, from a second sensor of the first device, aposition and/or an orientation of a tracking element configured to beattached to an ultrasound probe. In some embodiments, the second sensormay comprise a magnetometer, and the tracking element may comprise amagnet.

In some embodiments, the method may further comprise: receiving, from asecond sensor of the first device, at least one image of an environmentof the first device; and determining a position and/or an orientation ofan ultrasound probe based on the at least one image of the environmentof the first device.

In some embodiments, the first sensor may comprise a gyroscope.Alternatively or additionally, the method may further comprise:receiving, via a user control, a calibration of the first sensor of thefirst device.

Further aspects include a display device for simulated medical toolinsertion based on information received from a mobile device for use asa medical tool simulator. The display device may comprise at least oneprocessor configured to: receive, from the mobile device, an orientationof the mobile device; compute, based on the orientation of the mobiledevice, an orientation of a simulated medical tool; and display one ormore ultrasound images including an overlay indicating in real-time theorientation of the simulated medical tool.

In some embodiments, the at least one processor may be furtherconfigured to receive, from the mobile device, a first user inputrepresenting a depth of movement of the simulated medical tool, and theat least one processor may be configured to display the one or moreultrasound images at least by displaying the one or more ultrasoundimages including the overlay indicating in real-time the at least one ofthe orientation of the simulated medical tool and the first user input.

In some embodiments, the at least one processor may be furtherconfigured to: receive, from the mobile device, data indicating arelative position of the mobile device and an ultrasound probe.Additionally, the at least one processor may be further configured to:receive, from the mobile device, a position and/or an orientation of atracking element configured to be attached to the ultrasound probe,wherein the overlay may further indicate in real-time at least one ofthe position of the tracking element and the orientation of the trackingelement. Additionally, the at least one processor may be furtherconfigured to: receive, from the mobile device, a position of theultrasound probe and/or an orientation of the ultrasound probe, whereinthe overlay may further indicate in real-time at least one of theposition of the ultrasound probe and the orientation of the ultrasoundprobe.

In some embodiments, the at least one processor may be furtherconfigured to: receive, from an ultrasound probe, ultrasound data; anddisplay the one or more ultrasound images based on the ultrasound data.Alternatively or additionally, the at least one processor may be furtherconfigured to: receive, from a camera, at least one original ultrasoundimage captured by the camera; and display the one or more ultrasoundimages based on the at least one original ultrasound image.Additionally, the at least one processor may be further configured to:receive, via a user interface of the display device, a second userinput; and define, based on the second user input, an area of capturefor the camera.

In some embodiments, the one or more ultrasound images may comprise liveultrasound images. Alternatively or additionally, the one or moreultrasound images may comprise prerecorded ultrasound images.

In some embodiments, the at least one processor may be furtherconfigured to: compute the one or more ultrasound images based onprerecorded ultrasound images and an orientation of an ultrasound probe.

Additional aspects include a method of simulating a medical procedure.The method may comprise receiving information from a mobile deviceconfigured with an application to simulate a medical tool and totransmit orientation information; receiving ultrasound image data froman ultrasound probe; and computing an image of an object with an imageof a simulation oriented relative to the object based on the receivedinformation.

In some embodiments, computing the image may be based on an assumptionthat the mobile device has a known position relative to the ultrasoundprobe.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an exemplary apparatus for needle insertionsimulation according to some embodiments.

FIG. 2 is a diagram of an exemplary ultrasound probe according to someembodiments.

FIG. 3 is a diagram of an exemplary system for needle insertionsimulation according to some embodiments.

FIG. 4 is a diagram of an exemplary system for needle insertionsimulation for use on a subject according to some embodiments.

FIG. 5 is a diagram of an exemplary mobile device configured forsimulated needle insertion according to some embodiments.

FIG. 6 is a diagram of an exemplary system for needle insertionsimulation according to some embodiments.

FIG. 7 is a diagram of an exemplary system for needle insertionsimulation for use on a subject according to some embodiments.

FIG. 8 is a flowchart of a method for simulating needle insertionaccording to some embodiments.

FIG. 9 is a flowchart of an additional method for simulating needleinsertion according to some embodiments.

FIG. 10 is a flowchart of a method for simulating needle insertionaccording to some embodiments.

FIG. 11 is a flowchart of an additional method for simulating needleinsertion according to some embodiments.

FIG. 12 is a flowchart of an alternative method for simulating needleinsertion according to some embodiments.

FIG. 13 is a flowchart of a method for simulating needle insertion froma display device according to some embodiments.

FIG. 14 is a flowchart of an alternative method for simulating needleinsertion from a display device according to some embodiments.

FIG. 15 is a flowchart of another alternative method for simulatingneedle insertion from a display device according to some embodiments.

FIG. 16 is a flowchart of a method for simulating a medical procedureaccording to some embodiments.

FIG. 17 is a diagram illustrating a computer system on which someembodiments may be implemented.

DETAILED DESCRIPTION

The inventors have recognized and appreciated that more realistic,applicable, and accessible simulations of ultrasound-guided needleinsertion (and, therein, simulations of ultrasound usage itself) may beparticularly valuable because improper use of ultrasound and lack ofknowledge can increase the number of needle passes and movement inside apatient. Increased passes and internal movement of a needle may put thepatient at risk for mild complications, such as discomfort, as well asmore serious complications, like a pneumothorax. Moreover, improper useof or inexperience with ultrasound alone may cause additionaldiscomfort, inconvenience, and/or expense by prolonging the time thepatient must endure the potentially invasive procedure and the clinicalvisit.

The inventors have recognized and appreciated that a more realisticultrasound-guided needle insertion simulation may be provided using anactual living person as a subject rather than an inanimate object.Indeed, using a living person for the simulation may greatly improve thelearning experience for the user. Practicing needle insertion on aninanimate object may not provide a learning experience equivalent tousing a living person. For example, a living person may require a“bed-side manner,” may move at inopportune times, will at least move dueto breathing, may respond to the user in ways that are not easilysimulated by an inanimate object or even an animal, and so on.

The inventors have recognized and appreciated that a more realistic andapplicable ultrasound-guided needle insertion simulation may also beprovided using actual, live ultrasound images. Actual, live ultrasoundimages may provide more information and more variety of experience than,for example, prerecorded ultrasound images. Moreover, actual, liveultrasound images may allow the user to account for anatomicaldifferences and anomalies between individual people, which may requireimportant adjustments to be made for a successful needle insertion orpractice procedure. The actual, live ultrasound images may also allowthe user to overcome such anatomical differences or anomalies. Incontrast, many current simulators reference a library of pre-scanned andrecorded (or graphically modeled) ultrasound images (which may originatefrom a single scanned subject) or use real ultrasound machines to scanobjects that are not actually the subject of the injection. Such“fictitious” ultrasound images are not as realistic as actual, liveultrasound images and cannot account for anatomical differences andanomalies, although prerecorded ultrasound images may still be useful insome embodiments herein. For example, a simulation using prerecordedultrasound images may be less expensive than a system requiring actual,functioning ultrasound equipment.

The inventors have recognized and appreciated that a more realistic andapplicable ultrasound-guided needle insertion simulation may be providedby using an actual patient in the simulation immediately before theactual procedure on that same patient (in other words, in a“just-in-time” setting). For example, needle insertion can be verydifficult to perform in general, and the variation in anatomy betweendifferent patients (even when subtle) can increase the difficultysignificantly. A clinician may struggle to find a vein or avoid areas ororgans that the needle should not contact or enter, or may struggle toperform the operation well on one patient after having trained on adifferent subject (either via simulation or actual procedures). Thisdifficulty can create significant discomfort and pain for the patient aswell as unnecessary puncture wounds and bleeding or even more severeconditions, such as a pneumothorax. The inventors have recognized andappreciated that practicing the needle insertion using the actualpatient immediately before performing the actual insertion may enablethe clinician to more skillfully perform the procedure. For example, theclinician may use the simulated insertion to become familiar with thepatient's unique anatomy and learn the precise position and orientationthat the syringe should have to maximize the probability of a successfulinsertion. A clinician may practice the exact same needle passes justbefore placing a real needle inside a real patient. For example, if aclinician were getting ready to start an intravenous line on a patient,he or she could simulate the procedure on the patient in the samelocation on the arm.

The inventors have recognized and appreciated that a more realistic andapplicable ultrasound-guided needle insertion simulation may also beprovided by tracking the orientation and/or position of a needlesimulator (which may be a physical object or device that is used inplace of an actual syringe and needle). For example, tracking theorientation of a needle simulator may provide a simulation system withbasic information needed to represent to the user how well the simulatedneedle insertion is proceeding. One way in which this representation maybe performed is through display of an image representing the needlesimulator from a perspective not visible to the user. As the user canonly see the needle simulator from the perspective of his or her owneyes, for example, the user cannot see where the needle simulator wouldbe going inside a subject if the needle simulator were an actual needle.By tracking the orientation and/or position of the needle simulator anddisplaying an image of the inside of the subject with somerepresentation of the needle simulator (such as an overlay in the shapeof an actual needle) so that its simulated position inside the subject(if it were an actual needle) is apparent, the simulation can representto the user how well the simulated needle insertion is proceeding. Theimage or images of the inside of the subject may be actual, liveultrasound images (or prerecorded images, in some embodiments), asdiscussed herein. Therefore, without actually puncturing the subject'sskin, the user may see where an actual needle would go inside thesubject if placed in the same orientation and position as the needlesimulator. Moreover, the simulated needle insertion may more closelysimulate an actual ultrasound-guided needle insertion, in which a userwould see an actual needle inside the subject in ultrasound imagesduring the insertion. It should be appreciated that a needle simulatormay be a type of medical tool simulator.

The inventors have recognized and appreciated that a more realistic andapplicable ultrasound-guided needle insertion simulation may also beprovided by using magnetic tracking. For example, magnetic tracking candetermine the positions of object, or at least their relative positions,with sub-millimeter precision. Moreover, magnetic tracking can functionwithout requiring specific orientations and/or positions so that somedegree of line-of-sight is available, as optical sensors require. Forexample, optical sensors may be easily and accidentally blocked during aneedle insertion simulation, and their usability in such situations maybe severely impaired or non-existent. Magnetic sensors, on the otherhand, may provide the same usability regardless of their orientation,position, or obstructions. Moreover, magnetic sensors may provide higherprecision than optical sensors for comparable price-points.Additionally, commercially available magnetic sensors may be used, whichmay reduce the complexity and cost of the simulation system.

The inventors have recognized and appreciated that a more accessibleultrasound-guided needle insertion simulation may be provided in atraining or learning setting (such as a classroom) as well as a clinicalsetting. For example, even without variation in anatomy betweendifferent patients, needle insertion can be a very difficult procedureto learn. A clinician may need to practice many times before being ableto perform needle insertion (even with guidance from ultrasound) in away that is as fast, comfortable, and painless as possible for thepatient. Numerous practice insertions may allow the clinician to developmuscle memory that may be useful in all needle insertions.

The inventors have recognized and appreciated that a more accessibleultrasound-guided needle insertion simulation may be provided using adevice already in the possession of many consumers as a virtual needleor needle emulator. In some embodiments, a simulation may take advantageof a complex, capable mobile device that is commonly in the possessionof many potential users, such as clinical students. For example, asmartphone or other mobile device (which may not necessarily have acellular connection) may be used as a virtual needle by configuring themobile device with appropriate software (and, optionally, hardware).Using such mobile devices may reduce the cost of training due to theelimination of at least one item to rent or purchase. Moreover, usingsuch mobile devices may reduce the learning curve of users, as they mayalready be familiar with the interface of the mobile devices, which maybe used directly in the simulation.

The inventors have recognized and appreciated that a highly realisticultrasound-guided needle insertion simulation may be provided by using adevice that shares many or most physical characteristics with an actualsyringe and needle. For example, in some embodiments, an actual syringemay be fitted with a retractable “needle,” which may actually be anelongated member not made to puncture, with appropriate dimensions totake the place of an actual needle within a syringe. Alternatively, acustomized or modified syringe may be used with an included elongatedmember. Matching physical characteristics of an actual syringe andneedle, such as dimensions, weight, materials, and so on, may increasethe realism, precision, and utility of the simulation. Moreover, using adevice that is physically similar to an actual syringe and needle mayalso provide additional precision by providing more convenient locationsfor attaching or embedding one or more tracking devices that can berepresented realistically to the user, as described below. For example,while a mobile device such as a smartphone may provide a less expensiveand more accessible needle simulator, a device that is physicallysimilar to a syringe and needle may be able to support a tracking devicemore easily, and any representation of the tracking to the user may showsyringe and needle-shaped objects in a way that is faithful to theactual structure of the device.

It should be appreciated that embodiments and techniques herein mayalternatively or additionally be used for any invasive procedure insidea subject's body and are not limited to ultrasound-guided needleinsertion. For example, rather than a needle, any invasive medical toolmay be simulated, and rather than ultrasound, any suitable guidancesystem may be used.

It should also be appreciated that any embodiments or techniquesdescribed in one exemplary implementation herein may be used incombination with or as a replacement for any embodiments or techniquesdescribed in another exemplary implementation.

According to some embodiments, an ultrasound probe may be used to image(or pretend to image) a portion of an object (e.g., a patient or otherhuman subject), and a hand-held device (e.g., a tool or a mobile device)may be used to simulate an invasive medical tool (e.g., the device maybe made to look like the medical tool and simulate some function of themedical tool). Ultrasound data representing the object may be receivedfrom the ultrasound probe. Positional data (e.g., orientation and/orposition) may be received from the hand-held device and potentially fromthe ultrasound probe. The positional data may be used to calculatesimulated positional information of the hand-held device or a portionthereof that the medical tool would have if inserted into the objectfrom the orientation and/or position of the hand-held device or aportion thereof. An image of the object with an image of the hand-helddevice or a portion thereof positioned relative to the object may begenerated and displayed.

Exemplary Implementation of the System

FIG. 1 illustrates an exemplary apparatus for needle insertionsimulation according to some embodiments. In some embodiments, theapparatus may include a syringe 100. Additionally, the syringe 100 mayhave an injection end 110. The apparatus may also include an elongatedmember 120 protruding from the injection end 110 of the syringe 100.

According to some embodiments, the elongated member 120 may decrease alength of protrusion of the elongated member 120 from the injection end110 of the syringe 100. For example, the elongated member 120 mayretract into the syringe 100, in the direction from the injection end110 to the opposite end of the syringe 100, when a threshold force isapplied at an end 140 of the elongated member 120. The end 140 at whichthe threshold force may be applied may be located opposite the syringe100 when the elongated member 120 is fully protruding from the injectionend 110 of the syringe 100.

According to some embodiments, the retraction of the elongated member120 may occur when the end 140 of the elongated member 120 contacts asurface, such as human skin. For example, when a user begins a simulatedneedle insertion on a living subject, the elongated member 120 (whichmay simulate the needle itself) may be pressed against the skin of thesubject where the insertion is meant to occur. If the elongated member120 is pressed against the skin with a force at least as great as thethreshold force into the syringe 100, the elongated member 120 mayretract into the syringe 100 rather than puncturing the skin. Inaccordance with some embodiments the elongated member 120 may retractinto syringe 100 when it encounters a force that is less than requiredto puncture a sift surface, such as human skin, against which theelongated member might be pressed in operation. Setting the thresholdforce may make puncture unlikely. To make puncturing especiallyunlikely, the end 140 of the elongated member 120 may be flat or roundedrather than pointed, like many actual needles.

According to some embodiments, the syringe 100 or the elongated member120 may include a spring or any other suitable component (alternativelyor additionally, the syringe 100 or the elongated member 120 may betelescoping such that either or both may retract into themselves) thatmay maintain the elongated member 120 in a position of full protrusionout of the injection end 110 of the syringe 100 except when a thresholdforce is applied to the end 140 of the elongated member 120. When thethreshold force is no longer applied to the end 140 of the elongatedmember 120, the spring may return the elongated member 120 to theposition of full protrusion. For example, when a user is completing asimulated needle insertion, the user may move the syringe 100 away fromthe skin of the subject such that the force applied to the end 140 ofthe elongated member 120 into the syringe 100 is less than the thresholdforce. In response, the elongated member 120 may return to the positionof full protrusion.

According to some embodiments, the elongated member 120 may receive orsupport the attachment of at least one first sensor 130. Additionally,the first sensor 130 may indicate position information relating to theelongated member 130. Alternatively or additionally, the first sensor130 may indicate an orientation of the elongated member 130. Forexample, the first sensor 130 may be a magnetic sensor in a magnetictracking system (e.g., Ascension Flock of Birds), which may providesub-millimeter tracking precision. Alternatively, the first sensor 130may be a capacitive sensor, a potentiometer, or any other suitablesensor.

According to some embodiments, the first sensor 130 may be attached to atip (such as the end 140) of the elongated member 120. Additionally, thefirst sensor 130 may indicate information based on which anothercomponent may detect flexing of the elongated member 120 against anotherobject. For example, the first sensor 130 may indicate the orientationand position of the elongated member 120, based on which flexing may bedetermined by a magnetic tracking system or any suitable component. Forexample, if the end 140 of the elongated member 120 is not whereexpected (e.g., relative to a third sensor or another portion of theelongated member 120) given the orientation of the elongated member 120,this discrepancy may suggest flexing and may provide information neededto calculate a degree of flexing. The magnetic tracking system or othersuitable component may process data from the first sensor 130 or anothersensor that indicates strain or bending data. Alternatively oradditionally, the first sensor 130 may detect flexing directly. Forexample, the first sensor 130 may detect when the user is pushing theelongated member 120 against the skin of the subject such that flexingoccurs. Flexing may be a sign that the user is not performing asimulated insertion along the central axis of the syringe 100 and theelongated member 120.

FIG. 2 illustrates an exemplary ultrasound probe 200 according to someembodiments. In some embodiments, the ultrasound probe 200 may receiveor support the attachment of at least one second sensor 210.Additionally, the second sensor 210 may indicate position informationrelating to the ultrasound probe 200. Alternatively or additionally, thesecond sensor 210 may indicate an orientation of the ultrasound probe200.

According to some embodiments, the ultrasound probe 200 may capture oneor more images of a portion of an object against which the ultrasoundprobe 200 is positioned. Additionally, the portion of the object ofwhich images are captured may depend on the orientation and position ofthe ultrasound probe 200 with respect to the object. For example,rotating or translating the ultrasound probe 200 may cause a differentportion of the object to be captured than before the rotation ortranslation.

FIG. 3 illustrates an exemplary system for needle insertion simulationaccording to some embodiments. In some embodiments, the system mayinclude the syringe 100 and the ultrasound probe 200. The syringe 100and/or the elongated member 120 may receive or support the attachment ofat least one third sensor 150. The third sensor 150 may indicate arelative depth of movement of the elongated member 120 within thesyringe 100. For example, the third sensor 150 may indicate the degreeof protrusion of the elongated member 120 from the injection end 110 ofthe syringe 100 or from any other reference point. This information maybe used to show how deep inside the subject the elongated member 120would be if it were an actual needle. In other words, the relative depthmay represent motion of the needle (which may be displayed, as describedbelow).

According to some embodiments, the system may include a display device300. The display device 300 may be an ultrasound machine, a monitor, atelevision, a desktop or laptop computer, a smartphone, a tablet, and/orany other suitable components. The display device 300 may, in someembodiments, include at least one processor, at least one memory, atleast one computer-readable storage medium, and/or any other suitablecomponents. Alternatively or additionally, processing may occur inhardware associated with the display device 300 and/or in otherhardware, which may include at least one processor.

In some embodiments, the display device 300 may receive the position ofthe elongated member 120, the orientation of the elongated member 120,the position of the ultrasound probe 200, and/or the orientation of theultrasound probe 200. Alternatively, the display device 300 may receiveimages for display that are generated based on the position of theelongated member 120, the orientation of the elongated member 120, theposition of the ultrasound probe 200, and/or the orientation of theultrasound probe 200. Additionally, the display device 300 may receive,from the ultrasound probe 200, ultrasound data. For example, theultrasound data may include information needed to generate ultrasoundimages (and this generation of the ultrasound images may be performed bythe display device 300 or by any other suitable device) for display onthe display device 300.

According to some embodiments, the display device 300 may registerpositional information relating to the elongated member 120 and/or theultrasound probe 200 for use in computing simulated positionalinformation (e.g., a simulated orientation and/or position) of virtualrepresentations of the elongated member 120 and/or the ultrasound probe.For example, the display device 300 may compute a simulated position(based on the positional information) of the elongated member 120 withinone or more ultrasound images to be generated from the ultrasound data,which may be used to display ultrasound images with an overlay showingthe simulated position of the elongated member 120 (as described below).

According to some embodiments, the display device 300 may display, basedon the ultrasound data, one or more ultrasound images. Additionally, theultrasound images may include an overlay indicating, in real-time,information relating the elongated member 120 with the ultrasound probe200. Alternatively, the display device 300 may display such an overlayon top of or otherwise in combination with the ultrasound images. Forexample, the overlay may include a representation or depiction of theelongated member 120, which may be referred to as a simulated needle(which may be a virtual representation of an actual needle,corresponding to the needle simulator). The overlay may be stored orgenerated based on various sensors, including sensors 130, 150, and 210.It should be appreciated that a simulated needle may be a type ofsimulated medical tool.

According to some embodiments, the display device 300 (or any othersuitable component with processing ability) may compute the informationrelating the elongated member 120 with the ultrasound probe 200 based onat least one of the following: the position of the elongated member 120,the orientation of the elongated member 120, the relative depth ofmovement of the elongated member 120 within the syringe 100, theposition of the ultrasound probe 200, and the orientation of theultrasound probe 200. For example, the display device 300 may receivepositional information relating to the elongated member 120 and theultrasound probe 200 as well as ultrasound data, compute a simulatedposition (based on the positional information) of the elongated member120 within one or more ultrasound images to be generated from theultrasound data, and display the ultrasound images with the overlayshowing the simulated position of the elongated member 120. The relativedepth may be used to simulate what a user would see on ultrasound if areal needle were inserted to the indicated depth while actuallyperforming a procedure.

According to some embodiments, the orientation of the elongated member120 may include three degrees of freedom, including a yaw, a pitch, anda roll of the elongated member 120. Additionally, the position of theelongated member 120 may include three degrees of freedom, including anX position, a Y position, and a Z position (each corresponding to theirrespective axes). According to some embodiments, the depth of movementof the elongated member 120 within the syringe 100 may include onedegree of freedom.

According to some embodiments, the orientation of the ultrasound probe200 may include three degrees of freedom, including a yaw, a pitch, anda roll of the ultrasound probe 200. Additionally, the position of theultrasound probe 200 may include three degrees of freedom, including anX position, a Y position, and a Z position (each corresponding to theirrespective axes).

According to some embodiments, the ultrasound images may be or at leastinclude live ultrasound images, as discussed herein. Alternatively oradditionally, the ultrasound images may be or at least includeprerecorded ultrasound images. The advantages of both of these arediscussed herein.

According to some embodiments, the first sensor 130, the second sensor210, and the third sensor 150 may comprise magnetic sensors. Themagnetic sensors may be part of a magnetic tracking system such asAscension Flock of Birds, which may provide sub-millimeter trackingprecision. The sensors may be wired (although some sensors may bewireless, as described herein). Alternatively or additionally, thesesensors may be capacitive sensors, potentiometers, or any other suitablesensors. For example, the third sensor 150 may be a transducer, such asa linear potentiometer. In some embodiments, positional information fromthe first sensor 130, the second sensor 210, and/or the third sensor 150may be used to perform processing for simulating medical tool insertion,as described below.

FIG. 4 illustrates an exemplary system for needle insertion simulationfor use on a subject 400 according to some embodiments. In someembodiments, the system may include a needle simulator 100A (which maycorrespond to the syringe 100). The needle simulator 100A may include aprotruding member (which may correspond to a needle being simulated,while the rest of the needle simulator 100A may correspond to a syringebeing simulated). The protruding member may shorten or decrease thelength of its protrusion (shown inside the needle simulator 100A). Forexample, the needle simulator 100A may retract the protruding memberinto the needle simulator when a threshold force is applied at an end ofthe needle simulator 100A, as described herein. Additionally, the needlesimulator 100A may receive or support the attachment of at least onefirst sensor (which may correspond to first sensor 130) (not shown).

According to some embodiments, the system may include an ultrasoundprobe 200A (which may correspond to ultrasound probe 200). Theultrasound probe 200A may receive or support the attachment of at leastone second sensor (which may correspond to second sensor 210) (notshown). Additionally, the first sensor and/or the second sensor mayindicate a relative position between the needle simulator 100A and theultrasound probe 200A.

According to some embodiments, the first sensor and/or the second sensormay indicate a relative orientation between the needle simulator 100Aand the ultrasound probe 200A. Additionally, the needle simulator 100Aand/or the protruding member may receive or support the attachment of atleast one third sensor (which may correspond to third sensor 150) (notshown) configured to indicate a relative depth of movement of theprotruding member within the needle simulator 100A.

According to some embodiments, the system may include a display device300A (which may correspond to display device 300). The display device300A may receive the relative position of the needle simulator 100A andthe ultrasound probe 200A and/or the relative orientation of the needlesimulator 100A and the ultrasound probe 200A. Additionally, the displaydevice 300A may receive, from the ultrasound probe, ultrasound data, andmay display, based on the ultrasound data, one or more ultrasound imagesincluding an overlay indicating, in real-time, information relating theneedle simulator 100A with the ultrasound probe 200A.

According to some embodiments, the ultrasound probe 200A may be placedon the subject 400 in order to provide the ultrasound data received bythe display device 300A. Additionally, the needle simulator 100A may bepressed against the subject 400 (typically on the subject's skin, butclothing may also be used for a less realistic simulation) in order tosimulate the ultrasound-guided needle insertion. In some embodiments,the display device 300A may show the user (not shown) where the needlesimulator 100A would be inside the subject 400 if the needle simulator100A were an actual needle.

It should be appreciated from the foregoing that some embodiments aredirected to a method for simulating needle insertion, as illustrated inFIG. 8. The method may be performed by at least one processor caused todo so by executing instructions encoded on at least onecomputer-readable storage medium. The method begins at act 810, at whichultrasound data representing an object (e.g., the subject 400) beingimaged may be received. The method then proceeds to act 820, at whichposition data from or relating to a hand-held device (e.g., the syringe100) may be received.

The method proceeds then to act 830, at which a position of a needlesimulator (e.g., the elongated member 120) attached to the hand-helddevice may be computed. Then, the method proceeds to act 840, at whichan image of the object with an image of a simulated needle positionedrelative to the object based on the computed position of the needlesimulator may be generated. The method may then end or be repeated foradditional stages of a simulation or other simulations. The acts andtechniques of this method are described in further detail above.

It should be appreciated from the foregoing that some embodiments aredirected to an additional method for simulating needle insertion, asillustrated in FIG. 9. The method may be performed by at least oneprocessor caused to do so by executing instructions encoded on at leastone computer-readable storage medium. The method begins at act 810(previously described). The method then proceeds to act 820 (previouslydescribed). The method proceeds then to act 830 (previously described).Optionally, the method proceeds to act 833, at which the position of theneedle simulator, an orientation of the needle simulator, a position ofan ultrasound probe (e.g., ultrasound probe 200), and an orientation ofthe ultrasound probe may be received.

The method optionally proceeds to act 836, at which information relatingthe elongated member with the ultrasound probe may be computed based onat least one of the position of the elongated member, the orientation ofthe elongated member, the relative depth of movement of the elongatedmember within the syringe, the position of the ultrasound probe, and theorientation of the ultrasound probe. The method proceeds then to act 840(previously described). Then the method optionally proceeds to act 850,at which the image of the object with the image of the simulated needlepositioned relative to the object may be displayed based on theultrasound data, the image of the simulated needle comprising an overlayindicating in real-time the information relating the needle simulatorwith the ultrasound probe. The method may then end or be repeated foradditional stages of a simulation or other simulations. The acts andtechniques of this method are described in further detail above.

Additional Exemplary Implementation of the System

FIG. 5 illustrates an exemplary mobile device 500 for simulated needleinsertion according to some embodiments. The mobile device 500 mayinclude at least one processor (not shown), a display 510, at least onecontrol 520 (which may be solely on the display 510, which may be atouch screen), and at least one first sensor (not shown). The firstsensor may include an accelerometer and/or a gyroscope. Additionally,the mobile device may include at least one second sensor (not shown).The second sensor may include a magnetometer. In some embodiments, themobile device 500 may be a smartphone. Alternatively, the mobile device500 may be a tablet, PDA, a multimedia player, or any other suitabledevice.

According to some embodiments, the mobile device 500 may be configuredfor use as a needle simulator, as described herein. For example, themobile device 500 may download and/or install software (e.g., an “app”)that configures the mobile device for use as a needle simulator.

According to some embodiments, the mobile device 500 may present a userinterface 530 representing a simulated needle on the display 510 of themobile device 500. For example, the user interface 530 may include animage of a syringe and needle. The user interface 530 may include anindicator and control for user input representing a depth of movement ofthe simulated needle.

According to some embodiments, the mobile device 500 may receive, fromthe first sensor of the mobile device 500, an orientation of the mobiledevice 500. For example, the processor of the mobile device 500 may reador determine the mobile device's 500 orientation by checking a readingof the first sensor. Additionally, the mobile device 500 may transmitits orientation to a second device (not shown). This transmission may bemade via wired or wireless (e.g., Bluetooth) communication in any form.For example, the mobile device 500 may transmit the orientation to acomputer (e.g., a computer that is connected via wires or that is withinBluetooth range). Alternatively or additionally, if the mobile device500 has an internet connection, it may transmit the orientation to aserver (not shown), and the server may relay the orientation to thesecond device. In some embodiments, the second device may include adisplay device (not shown), as illustrated in FIGS. 6 and 7. The displaydevice may use the orientation to compute how to represent the mobiledevice 500 (e.g., how to display a virtual needle) in an image to bedisplayed, as described below.

FIG. 6 illustrates an exemplary system for needle insertion simulationaccording to some embodiments. In some embodiments, the system mayinclude the mobile device 500 and the ultrasound probe 200. In someembodiments, the system may include the display device 300.

FIG. 7 illustrates an exemplary system for needle insertion simulationfor use on a subject 400 according to some embodiments. In someembodiments, the system may include the mobile device 500, theultrasound probe 200, and/or the display device 300. The followingdiscussion may refer to FIGS. 6 and 7 jointly, with some elements notexplicitly labeled in both FIGS. 6 and 7.

According to some embodiments, the mobile device 500 may present a userinterface 530 representing a simulated needle on the display 510 of themobile device 500, as discussed herein.

According to some embodiments, the mobile device 500 may receive acalibration of the first sensor of the mobile device 500 from a usercontrol (e.g., control 520) or interface, which may be operated by theuser. For example, when the user has placed the mobile device 500 at thedesired position and/or orientation for the simulation, the user maytrigger calibration of the first sensor of the mobile device 500 (forexample, via control 520 or user interface 530). This way, the systemmay treat the position and/or orientation of the mobile device 500 atthe time of calibration as the starting or “zero” position and/ororientation from which later movement is measured. In some embodiments,the user must maintain the starting position and only alter theorientation of the mobile device 500, which the system may assumeoccurs.

Alternatively, this calibration may be made unnecessary by tracking theposition of the mobile device 500, which may be accomplished by trackingthe position of the ultrasound probe 200 relative to the mobile device500. According to some embodiments, the ultrasound probe 200 may receiveor support the attachment of at least one tracking element 220. Thetracking element 220 may be tracked by the second sensor of the mobiledevice 500, providing to the mobile device 500 (which may receive) theorientation and/or the position of the tracking element 220 and,thereby, the ultrasound device 200 relative to the second sensor (andthereby the mobile device 500). In some embodiments, the trackingelement 220 may be a magnet, and, as discussed, the second sensor of themobile device 500 may be a magnetometer. The magnet may be less thanabout five percent of the size of the ultrasound probe 200 and may beattached to the ultrasound probe 200 using a clip, tape, or any othersuitable attachment mechanism. The inventors have recognized andappreciated that tracking the position and/or orientation of theultrasound probe 200 relative to the mobile device 500 may provide moreprecision and realism to a simulation and potentially prevent a need forcalibration of the position and/or orientation of the mobile device 500,as discussed herein. Moreover, this tracking of the ultrasound probe 200relative to the mobile device 500 may provide a more robust tracking ofthe mobile device 500 than merely using the first sensor of the mobiledevice 500, and the user may reposition the mobile device 500 withoutneeding to perform calibration again.

Another potential alternative to calibration may include (althoughcalibration may be used as well), according to some embodiments, themobile device 500 receiving, from a second sensor (not shown) of themobile device 500, at least one image of an environment of the mobiledevice 500. The second sensor may include a camera, another opticalsensor, or any other suitable sensor able to produce an image. Themobile device 500 may determine a position and/or an orientation of theultrasound probe 200 and/or the subject 400 (e.g., relative to themobile device 500) based on the image of the environment of the mobiledevice 500. For example, the processor of the mobile device 500 may useimage processing to estimate position and/or orientation information ofobjects in the image.

According to some embodiments, the display device 300 may receive theorientation of the mobile device 500, as discussed above with regard tothe second device. Alternatively or additionally, the display device 300may receive the position of the mobile device 500, the position of theultrasound probe 200, and/or the orientation of the ultrasound probe200. The display device 300 may alternatively or additionally receive,from the mobile device 500, data indicating a relative position of themobile device 500 and the ultrasound probe 200. For example, the displaydevice 300 may receive, from the mobile device 500, a position and/or anorientation of the tracking element 220 attached to the ultrasound probe200.

Alternatively, the display device 300 may receive images for displaythat are generated based on the position of the mobile device 500, theorientation of the mobile device 500, the position of the ultrasoundprobe 200, and/or the orientation of the ultrasound probe 200. Forexample, if the display device 300 is merely a display monitor orscreen, it may only receive images or an image signal from anotherdevice.

According to some embodiments, any of the positional information(including orientation) of any components may be calculated (e.g., bythe processor of the mobile device 500 or by the processor of thedisplay device 300) based on any of the other position information givenrelationships. For example, the orientation of the mobile device 500 maybe calculated based on (or the precision of the orientation may beimproved based on) the relative orientation between the mobile device500 and the ultrasound probe 200.

According to some embodiments, the orientation of the mobile device 500may include three degrees of freedom, including a yaw, a pitch, and aroll of the mobile device 500. Additionally, the position of theultrasound probe 200 (where available) may include three degrees offreedom, including an X position, a Y position, and a Z position (eachcorresponding to their respective axes).

According to some embodiments, the orientation of the ultrasound probe200 may include three degrees of freedom, including a yaw, a pitch, anda roll of the ultrasound probe 200. Additionally, the position of theultrasound probe 200 may include three degrees of freedom, including anX position, a Y position, and a Z position (each corresponding to theirrespective axes).

According to some embodiments, the ultrasound probe 200 may be placed onthe subject 400 in order to provide the ultrasound data received by thedisplay device 300. Additionally, the mobile device 500 may be pressedagainst the subject 400 (typically on the subject's skin, but clothingmay also be used for a less realistic simulation) in order to prepare tosimulate the ultrasound-guided needle insertion. In some embodiments,the display device 300 may show the user (not shown) where the simulatedneedle would be inside the subject 400 if the needle simulator were anactual needle.

According to some embodiments, the display device 300 may receive, fromthe ultrasound probe 200, ultrasound data. For example, as theultrasound probe 200 scans the subject, the ultrasound probe 200 mayrelay ultrasound data from the scanning to the display device 300. Theultrasound data may include information needed to generate ultrasoundimages (and this generation of the ultrasound images may be performed bythe display device 300 or by any other suitable device) for display onthe display device 300.

According to some embodiments, the ultrasound images may be or at leastinclude live ultrasound images, as discussed herein. Alternatively oradditionally, the ultrasound images may be or at least includeprerecorded ultrasound images. The advantages of both of these arediscussed herein.

According to some embodiments, the display device 300 may receive, froma camera, at least one original ultrasound image captured by the camera.For example, if the ultrasound images (whether prerecorded or live) arenot easily accessible to the display device 300 (or for any otherreason), a camera may be used to record the ultrasound images from anactual display that is displaying them, such as a ultrasound systemmanufacturer's standard monitor (or other pre-existing display). Then,the captured ultrasound images may be used and/or modified in wayssimilar to those described with respect to other ultrasound imagesherein. For example, as described below, the ultrasound images may bedisplayed on the display device 300 based on the original ultrasoundimage(s). In some embodiments, the camera may include a “webcam,” acamera module of the display device 300 (such as a built-in orattachable camera), an analog camera, or any other suitable opticaldevice. Alternatively, original ultrasound image(s) may be capturedusing a video capture device (not shown) that may be connected to theactual display, such as via an external display pot like HDMI or VGA.

According to some embodiments, the display device 300 may include a userinterface via which the display device 300 may receive a second userinput. The display device 300 may then define, based on the second userinput, an area of capture for the camera. For example, if the camera ispositioned, zoomed, or focused such that the camera captures objectsthat are not relevant to the ultrasound images, such as backgroundenvironment (e.g., curtains, walls, and so on) outside of amanufacturer's standard monitor, the user may input bounds to the areaof capture so that only the desired area is provided to the displaydevice 300. Additional processing, such as displaying an overlay, may belimited to this bounded or cropped area of the images. In someembodiments, the display device 300 may receive input via a mouse, akeyboard, and/or a touch pad that defines the area of capture using oneor more single-point operations, a dragging operation, or any othersuitable operation.

According to some embodiments, the camera may be mounted on the actualdisplay. For example, the camera may be attached to the actual displayusing a clip, temporary adhesive, suction, or any other suitablemounting mechanism or technique. In some embodiments, the angle of thecamera relative to the actual display may be fixed so that they areparallel. Alternatively, the angle may be adjustable.

Alternatively, the display device 300 may be the actual display (e.g.,the manufacturer's standard monitor), and the system may interfacedirectly with the display device 300 and/or components provided imagesto the display device 300. For example, native or modified code withinthe manufacturer's equipment could be used to manipulate ultrasoundimages for the additional processing described herein, such asdisplaying an overlay.

According to some embodiments, the mobile device 500 may receive, viathe user interface 530, a user input representing a depth of movement ofthe simulated needle. The user input may be a sliding user input. Forexample, the user may swipe (on, e.g., the touch screen of the mobiledevice 500) from the back of a representation of a needle to the fronton the user interface 530 to simulate a needle insertion. Additionally,the mobile device may transmit the user input to the second device,which may be the display device 300.

According to some embodiments, the display device 300 may display, basedon the ultrasound data, one or more ultrasound images. Additionally, theultrasound images may include an overlay (e.g., the display device 300may display the overlay on top of or otherwise in combination with theultrasound images). The overlay may indicate, in real-time, theorientation of the mobile device 500. For example, the display device300 may compute, based on the orientation of the mobile device 500, anorientation of the simulated needle, and the overlay may indicate theorientation of the simulated needle, thereby indicating the orientationof the mobile device 500. Alternatively or additionally, the overlay mayindicate information relating the mobile device 500 with the ultrasoundprobe 200. For example, the overlay may include a representation ordepiction of the simulated needle (which may be similar to or differentfrom the simulated needle displayed on the mobile device 500). Therepresentation of the simulated needle may be positioned and oriented inthe images such that, as the user moves the mobile device 500, therepresentation moves accordingly within the images. The overlay may bestored for repeat use, or the overlay may be generated based oninformation received from the mobile device 500.

According to some embodiments, the overlay may indicate, in real-time,the user input received from the mobile device 500. For example, whenthe user performs a sliding user input to simulate needle insertion(e.g., by swiping up on the user interface 530), the user input may betransmitted to the display device 300 and indicated by the overlay, suchas by displaying an extending of the representation of the simulatedneedle to represent a needle extending from a syringe.

According to some embodiments, the overlay may indicate, in real-time,the position of the tracking element 220 (and thereby of the ultrasoundprobe 200) and/or the orientation of the tracking element 220 (andthereby of the ultrasound probe 200). For example, if the ultrasoundimages displayed are prerecorded, showing positional informationrelating to the tracking element 220 (and thereby the ultrasound probe200) may be useful because the ultrasound images may not change as aresult of moving the ultrasound probe 200. However, even prerecordedimages may change if they were taken or generated from differentperspectives. In some embodiments, for example, the display device 300may compute or alter ultrasound images for display based on prerecordedultrasound images and an orientation and/or position of the ultrasoundprobe 200. Alternatively or additionally, the display device 300 mayselect portions of prerecorded ultrasound images based on theorientation and/or position of the ultrasound probe 200.

As discussed above, it should be appreciated that embodiments andtechniques herein may alternatively or additionally be used for anyother invasive procedure inside a subject 400. For example, according tosome embodiments, the display device 300 may receive information from amobile device 500 configured with an application to simulate a medicaltool and to transmit orientation information. Additionally, the displaydevice 300 may receive ultrasound image data (or any other suitabledata) from an ultrasound probe 200. Additionally, the display device 300may compute an image of an object (e.g., subject 400) with an image of asimulation oriented relative to the object based on the receivedinformation. In some embodiments, computing the image may be based on anassumption that the mobile device 500 has a known position relative tothe ultrasound probe 200. For example, by assuming that the mobiledevice 500 has a known position relative to the ultrasound probe 200,computing the image may be shorter than alternative embodiments, wherethe relative positions of the ultrasound probe 200 and the mobile device500 within the same frame of reference may be used to position the imageof the simulation relative to the image of the object.

It should be appreciated from the foregoing that some embodiments aredirected to a method for simulating needle insertion, as illustrated inFIG. 10. The method may be performed by at least one processor caused todo so by executing instructions encoded on at least onecomputer-readable storage medium. The method begins at act 1010, atwhich a first device (e.g., the mobile device 500) may be configured foruse as a needle simulator. The method then proceeds to act 1020, atwhich a user interface representing a simulated needle on a display(e.g., display 510) of the first device may be presented.

The method proceeds then to act 1030, at which an orientation of thefirst device may be received from a first sensor (e.g., an accelerometerand/or gyroscope) of the first device. Then, the method proceeds to act1040, at which the orientation of the first device may be transmitted toa second device (e.g., display device 300). The method may then end orbe repeated for additional stages of a simulation or other simulations.The acts and techniques of this method are described in further detailabove.

It should be appreciated from the foregoing that some embodiments aredirected to an additional method for simulating needle insertion, asillustrated in FIG. 11. The method may be performed by at least oneprocessor caused to do so by executing instructions encoded on at leastone computer-readable storage medium. The method begins at act 1010(previously described). The method then proceeds to act 1020 (previouslydescribed). The method proceeds then to act 1030 (previously described).Then, the method optionally proceeds to act 1035, at which a positionand/or an orientation of a tracking element (e.g., tracking element 220)configured to be attached to an ultrasound probe (e.g., ultrasound probe200) may be received from a second sensor (e.g., a magnetometer) of thefirst device.

The method proceeds then to act 1040 (previously described). Then,optionally, the method may proceed to act 1045, at which a sliding userinput representing a depth of movement of the simulated needle may bereceived via the user interface. The method then proceeds optionally toact 1050, at which the sliding user input may be transmitted to thesecond device. The method may then end or be repeated for additionalstages of a simulation or other simulations. The acts and techniques ofthis method are described in further detail above.

It should be appreciated from the foregoing that some embodiments aredirected to an alternative method for simulating needle insertion, asillustrated in FIG. 12. The method may be performed by at least oneprocessor caused to do so by executing instructions encoded on at leastone computer-readable storage medium. The method begins at act 1010(previously described). The method then proceeds to act 1020 (previouslydescribed).

The method proceeds then, optionally, to act 1023, at which acalibration of the first sensor of the first device may be received.Then, the method optionally proceeds to act 1026, at which at least oneimage of an environment of the first device may be received from asecond sensor (e.g., a camera) of the first device.

The method optionally proceeds then to act 1029, at which a positionand/or an orientation of an ultrasound probe (e.g., ultrasound probe200) based on the at least one image of the environment of the firstdevice may be determined. Then the method proceeds to act 1030(previously described). The method then proceeds to act 1040 (previouslydescribed). The method may then end or be repeated for additional stagesof a simulation or other simulations. The acts and techniques of thismethod are described in further detail above.

It should be appreciated from the foregoing that some embodiments aredirected to a method for simulating needle insertion from a displaydevice, as illustrated in FIG. 13. The method begins at act 1310, atwhich a mobile device (e.g., the mobile device 500) may be configuredfor use as a needle simulator. The method then proceeds to act 1320, atwhich an orientation of the mobile device may be received from themobile device. The method then proceeds to act 1325, at which anorientation of a simulated needle may be computed based on theorientation of the mobile device.

The method optionally proceeds then to act 1330, at which ultrasounddata may be received from an ultrasound probe (e.g., ultrasound probe200). Then, the method optionally proceeds to act 1340, at which dataindicating a relative position of the mobile device and an ultrasoundprobe may be received from the mobile device.

The method proceeds then, optionally, to act 1350, at which a positionand/or an orientation of a tracking element (e.g., a magnet) configuredto be attached to the ultrasound probe may be received from the mobiledevice. Then, optionally, the method may proceed to act 1360, at which afirst user input representing a depth of movement of a simulated needlemay be received from the mobile device.

The method then proceeds to act 1370, at which one or more ultrasoundimages including an overlay indicating in real-time the orientation ofthe simulated needle may be displayed. The method may then end or berepeated for additional stages of a simulation or other simulations. Theacts and techniques of this method are described in further detailabove.

It should be appreciated from the foregoing that some embodiments aredirected to an alternative method for simulating needle insertion from adisplay device, as illustrated in FIG. 14. The method begins at act 1310(previously described). The method then proceeds to act 1320 (previouslydescribed). The method then proceeds to act 1325 (previously described).

The method optionally proceeds then to act 1335, at which a second userinput may be received via a user interface of a display device (e.g.,display device 300). Then, the method optionally proceeds to act 1345,at which an area of capture for the camera may be defined based on thesecond user input.

The method proceeds then, optionally, to act 1355, at which at least oneoriginal ultrasound image captured by a camera may be received from thecamera. The method then proceeds to act 1370 (previously described). Themethod may then end or be repeated for additional stages of a simulationor other simulations. The acts and techniques of this method aredescribed in further detail above.

It should be appreciated from the foregoing that some embodiments aredirected to another alternative method for simulating needle insertionfrom a display device, as illustrated in FIG. 15. The method begins atact 1310 (previously described). The method then proceeds to act 1320(previously described). The method optionally proceeds then to act 1365,at which one or more ultrasound images based on prerecorded ultrasoundimages and an orientation of an ultrasound probe (e.g., ultrasound probe200) may be computed. The method then proceeds to act 1370 (previouslydescribed). The method may then end or be repeated for additional stagesof a simulation or other simulations. The acts and techniques of thismethod are described in further detail above.

It should be appreciated from the foregoing that some embodiments aredirected to a method for simulating a medical procedure, as illustratedin FIG. 16. The method begins at act 1610, at which information from amobile device configured with an application to simulate a medical tooland to transmit orientation information may be received.

The method then proceeds to act 1620, at which ultrasound image datafrom a probe may be received. The method proceeds then to act 1630, atwhich an image of an object with an image of a simulation orientedrelative to the object based on the received information may becomputed. The method may then end or be repeated for additional stagesof a simulation or other simulations. The acts and techniques of thismethod are described in further detail above.

Computing Environment

Techniques as described herein may be implemented on any suitablehardware, including a programmed computing system. For example,calculation of positional information of a needle simulator may beperformed by programming a computing device. Similarly, manipulation ofultrasound images may be performed by a programmed computing device.FIGS. 3 through 7 illustrate components and systems that may beimplemented with multiple computing devices, which may be distributedand/or centralized. Also, FIGS. 8 through 16 illustrate processes thatmay include algorithms executing on at least one computing device. FIG.17 illustrates an example of a suitable computing system environment1700 on which embodiments of these algorithms may be implemented. Thiscomputing system may be representative of a computing system thatimplements the techniques described herein. However, it should beappreciated that the computing system environment 1700 is only oneexample of a suitable computing environment and is not intended tosuggest any limitation as to the scope of use or functionality of theinvention. Neither should the computing environment 1700 be interpretedas having any dependency or requirement relating to any one orcombination of components illustrated in the exemplary operatingenvironment 1700.

The invention is operational with numerous other computing systemenvironments or configurations configured to perform the functionsdescribed herein. Examples of well-known computing systems,environments, and/or configurations that may be suitable for use withthe invention include, but are not limited to, personal computers,server computers, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputers, mainframe computers,distributed computing environments or cloud-based computing environmentsthat include any of the above systems or devices, and the like.

The computing environment may execute computer-executable instructions,such as program modules. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 17, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 310. Though a programmed general purpose computer isillustrated, it should be understood by one of skill in the art thatalgorithms may be implemented in any suitable computing device.Accordingly, techniques as described herein may be implemented in anysuitable system. These techniques may be implemented in such networkdevices as originally manufactured or as a retrofit, such as by changingprogram memory devices holding programming for such network devices orsoftware download. Thus, some or all of the components illustrated inFIG. 17, though illustrated as part of a general purpose computer, maybe regarded as representing portions of a node or other component in anetwork system.

Components of computer 310 may include, but are not limited to, aprocessing unit 320, a system memory 330, and a system bus 321 thatcouples various system components including the system memory 330 to theprocessing unit 320. The system bus 321 may be any of several types ofbus structures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. By wayof example and not limitation, such architectures include IndustryStandard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus,Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA)local bus, and Peripheral Component Interconnect (PCI) bus also known asMezzanine bus.

Computer 310 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 310 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store the desired informationand that can be accessed by computer 310. Communication media typicallyembodies computer readable instructions, data structures, programmodules, or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency (RF), infrared (IR), and other wireless media. Combinations ofany of the above should also be included within the scope of computerreadable media.

The system memory 330 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 331and random access memory (RAM) 332. A basic input/output system 333(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 310, such as during start-up, istypically stored in ROM 331. RAM 332 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 320. By way of example and notlimitation, FIG. 17 illustrates operating system 334, applicationprograms 335, other program modules 336, and program data 337.

The computer 310 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 17 illustrates a hard disk drive 341 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 351that reads from or writes to a removable, nonvolatile magnetic disk 352,and an optical disk drive 355 that reads from or writes to a removable,nonvolatile optical disk 356 such as a CD-ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 341 is typically connectedto the system bus 321 through a non-removable memory interface such asinterface 340, and magnetic disk drive 351 and optical disk drive 355are typically connected to the system bus 321 by a removable memoryinterface, such as interface 350.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 17, provide storage of computer readableinstructions, data structures, program modules, and other data for thecomputer 310. In FIG. 17, for example, hard disk drive 341 isillustrated as storing operating system 344, application programs 345,other program modules 346, and program data 347. Note that thesecomponents can either be the same as or different from operating system334, application programs 335, other program modules 336, and programdata 337. Operating system 344, application programs 345, other programmodules 346, and program data 347 are given different numbers here toillustrate that, at a minimum, they are different copies. A user mayenter commands and information into the computer 310 through inputdevices such as a keyboard 362 and pointing device 361, commonlyreferred to as a mouse, trackball, or touch pad. Other input devices(not shown) may include a microphone, joystick, game pad, satellitedish, scanner, or the like. These and other input devices are oftenconnected to the processing unit 320 through a user input interface 360that is coupled to the system bus, but may be connected by otherinterface and bus structures, such as a parallel port, game port, or auniversal serial bus (USB). A monitor 391 or other type of displaydevice is also connected to the system bus 321 via an interface, such asa video interface 390. In addition to the monitor, computers may alsoinclude other peripheral output devices such as speakers 397 and printer396, which may be connected through an output peripheral interface 395.

The computer 310 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer380. The remote computer 380 may be a personal computer, a server, arouter, a network PC, a peer device, or some other common network node,and typically includes many or all of the elements described aboverelative to the computer 310, although only a memory storage device 381has been illustrated in FIG. 17. The logical connections depicted inFIG. 17 include a local area network (LAN) 371 and a wide area network(WAN) 373, but may also include other networks. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets, and the Internet.

When used in a LAN networking environment, the computer 310 is connectedto the LAN 371 through a network interface or adapter 370. When used ina WAN networking environment, the computer 310 typically includes amodem 372 or other means for establishing communications over the WAN373, such as the Internet. The modem 372, which may be internal orexternal, may be connected to the system bus 321 via the user inputinterface 360, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 310, orportions thereof, may be stored in the remote memory storage device. Byway of example and not limitation, FIG. 17 illustrates remoteapplication programs 385 as residing on memory device 381. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

Having thus described several aspects of at least one embodiment of thisinvention, it is to be appreciated that various alterations,modifications, and improvements will readily occur to those skilled inthe art.

Such alterations, modifications, and improvements are intended to bepart of this disclosure, and are intended to be within the spirit andscope of the invention. Further, though advantages of the presentinvention are indicated, it should be appreciated that not everyembodiment of the invention will include every described advantage. Someembodiments may not implement any features described as advantageousherein and in some instances. Accordingly, the foregoing description anddrawings are by way of example only.

The above-described embodiments of the present invention can beimplemented in any of numerous ways. For example, the embodiments may beimplemented using hardware, software or a combination thereof. Whenimplemented in software, the software code can be executed on anysuitable processor or collection of processors, whether provided in asingle computer or distributed among multiple computers. Such processorsmay be implemented as integrated circuits, with one or more processorsin an integrated circuit component. Though, a processor may beimplemented using circuitry in any suitable format.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device not generally regarded as acomputer but with suitable processing capabilities, including a PersonalDigital Assistant (PDA), a smart phone or any other suitable portable orfixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including as a local area network or a wide area network,such as an enterprise network or the Internet. Such networks may bebased on any suitable technology and may operate according to anysuitable protocol and may include wireless networks, wired networks orfiber optic networks.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone of a variety of operating systems or platforms. Additionally, suchsoftware may be written using any of a number of suitable programminglanguages and/or programming or scripting tools, and also may becompiled as executable machine language code or intermediate code thatis executed on a framework or virtual machine.

In this respect, the invention may be embodied as a computer readablestorage medium (or multiple computer readable media) (e.g., a computermemory, one or more floppy discs, compact discs (CD), optical discs,digital video disks (DVD), magnetic tapes, flash memories, circuitconfigurations in Field Programmable Gate Arrays or other semiconductordevices, or other tangible computer storage medium) encoded with one ormore programs that, when executed on one or more computers or otherprocessors, perform methods that implement the various embodiments ofthe invention discussed above. As is apparent from the foregoingexamples, a computer readable storage medium may retain information fora sufficient time to provide computer-executable instructions in anon-transitory form. Such a computer readable storage medium or mediacan be transportable, such that the program or programs stored thereoncan be loaded onto one or more different computers or other processorsto implement various aspects of the present invention as discussedabove. As used herein, the term “computer-readable storage medium”encompasses only a computer-readable medium that can be considered to bea manufacture (i.e., article of manufacture) or a machine. Alternativelyor additionally, the invention may be embodied as a computer readablemedium other than a computer-readable storage medium, such as apropagating signal.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of the present invention asdiscussed above. Additionally, it should be appreciated that accordingto one aspect of this embodiment, one or more computer programs thatwhen executed perform methods of the present invention need not resideon a single computer or processor, but may be distributed in a modularfashion amongst a number of different computers or processors toimplement various aspects of the present invention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

Also, data structures may be stored in computer-readable media in anysuitable form. For simplicity of illustration, data structures may beshown to have fields that are related through location in the datastructure. Such relationships may likewise be achieved by assigningstorage for the fields with locations in a computer-readable medium thatconveys relationship between the fields. However, any suitable mechanismmay be used to establish a relationship between information in fields ofa data structure, including through the use of pointers, tags or othermechanisms that establish relationship between data elements.

Various aspects of the present invention may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Also, the invention may be embodied as a method, of which an example hasbeen provided. The acts performed as part of the method may be orderedin any suitable way. Accordingly, embodiments may be constructed inwhich acts are performed in an order different than illustrated, whichmay include performing some acts simultaneously, even though shown assequential acts in illustrative embodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

In the attached claims, various elements are recited in differentclaims. However, the claimed elements, even if recited in separateclaims, may be used together in any suitable combination.

1. A medical tool insertion simulation apparatus comprising: a syringehaving an injection end; and an elongated member protruding from theinjection end of the syringe, the elongated member being configured todecrease a length of protrusion of the elongated member from theinjection end of the syringe, the elongated member being configured toreceive at least one first sensor configured to indicate positioninformation relating to the elongated member.
 2. The medical toolinsertion simulation apparatus of claim 1, further comprising: anultrasound probe configured to receive at least one second sensor toindicate a position of the ultrasound probe.
 3. The medical toolinsertion simulation apparatus of claim 2, wherein: the at least onefirst sensor is further configured to indicate an orientation of theelongated member, and the at least one second sensor is furtherconfigured to indicate an orientation of the ultrasound probe.
 4. Themedical tool insertion simulation apparatus of claim 3, wherein: theapparatus is configured to receive at least one third sensor configuredto indicate a relative depth of movement of the elongated member withinthe syringe.
 5. The medical tool insertion simulation apparatus of claim4, further comprising: a display device configured to: receive theposition of the elongated member, the orientation of the elongatedmember, the position of the ultrasound probe, and the orientation of theultrasound probe; receive, from the ultrasound probe, ultrasound data;and display, based on the ultrasound data, one or more ultrasound imagesincluding an overlay indicating in real-time information relating theelongated member with the ultrasound probe.
 6. The medical toolinsertion simulation apparatus of claim 5, wherein: the display deviceis further configured to compute the information relating the elongatedmember with the ultrasound probe based on at least one of the positionof the elongated member, the orientation of the elongated member, therelative depth of movement of the elongated member within the syringe,the position of the ultrasound probe, and the orientation of theultrasound probe.
 7. The medical tool insertion simulation apparatus ofclaim 5, wherein: the one or more ultrasound images comprise liveultrasound images.
 8. The medical tool insertion simulation apparatus ofclaim 2, wherein: the at least one first sensor and the at least onesecond sensor comprise magnetic sensors.
 9. The medical tool insertionsimulation apparatus of claim 1, wherein: the at least one first sensoris attached to a tip of the elongated member and is further configuredto output information indicating flexing of the elongated member againstanother object.
 10. The medical tool insertion simulation apparatus ofclaim 1, wherein: the elongated member is configured to decrease thelength of protrusion of the elongated member from the injection end ofthe syringe by retracting into the syringe when a threshold force isapplied at an end of the elongated member that is opposite the syringewhen the elongated member is fully protruding from the injection end ofthe syringe.
 11. At least one computer-readable storage medium encodedwith executable instructions that, when executed by at least oneprocessor, cause the at least one processor to perform a method forsimulating medical tool insertion, the method comprising: receivingultrasound data representing an object being imaged; receiving positiondata from a hand-held device; computing a position of a medical toolsimulator attached to the hand-held device; and generating an image ofthe object with an image of a simulated medical tool positioned relativeto the object based on the computed position of the medical toolsimulator.
 12. The at least one computer-readable storage medium ofclaim 11, wherein the method further comprises: receiving an orientationof the medical tool simulator, a position of an ultrasound probe, and anorientation of the ultrasound probe; and displaying, based on theultrasound data, the image of the object with the image of the simulatedmedical tool positioned relative to the object, the image of thesimulated medical tool comprising an overlay indicating in real-timeinformation relating the medical tool simulator with the ultrasoundprobe.
 13. The at least one computer-readable storage medium of claim12, further comprising: computing the information relating the medicaltool simulator with the ultrasound probe based on at least one of theposition of the medical tool simulator, the orientation of the medicaltool simulator, the position of the ultrasound probe, and theorientation of the ultrasound probe.
 14. The at least onecomputer-readable storage medium of claim 13, wherein: the image of theobject comprises one or more live ultrasound images.
 15. A medical toolinsertion simulation system comprising: a medical tool simulatorconfigured to decrease a length of protrusion of a protruding member ofthe medical tool simulator; and an ultrasound probe, wherein: themedical tool simulator is configured to receive at least one firstsensor, the ultrasound probe is configured to receive at least onesecond sensor, and the at least one first sensor and/or the at least onesecond sensor is configured to indicate a relative position between themedical tool simulator and the ultrasound probe.
 16. The medical toolinsertion simulation system of claim 15, wherein: the at least one firstsensor and/or the at least one second sensor is further configured toindicate a relative orientation between the medical tool simulator andthe ultrasound probe.
 17. The medical tool insertion simulation systemof claim 16, wherein: the medical tool simulator and/or the protrudingmember is configured to receive at least one third sensor configured toindicate a relative depth of movement of the protruding member withinthe medical tool simulator.
 18. The medical tool insertion simulationsystem of claim 16, further comprising: a display device configured to:receive the relative position of the medical tool simulator and theultrasound probe and the relative orientation of the medical toolsimulator and the ultrasound probe; receive, from the ultrasound probe,ultrasound data; and display, based on the ultrasound data, one or moreultrasound images including an overlay indicating in real-timeinformation relating the medical tool simulator with the ultrasoundprobe.
 19. The medical tool insertion simulation system of claim 15,wherein: the medical tool simulator is configured to decrease the lengthof protrusion of the protruding member by retracting the protrudingmember into the medical tool simulator when a threshold force is appliedat an end of the medical tool simulator.